I am not sure why CI is failing. When I run the tests locally they pass.

Force the tests to re-run (unsure how), and see if they still fail? If they do, force a re-run on the base (unsure how) and see if they still pass? If all that is true, then dig deeper.

The failure we get is genuine. I think you can get it too by running both test_verify.py and test_vgen.py in the same run. Then it tries to load a library made in the test_verify mode while assuming it was made in the test_vgen mode, or something.

When I run the tests locally the cffi.verifier.cleanup_tmpdir() call cleans out the old module. I am not sure why in this test only the cleanup does not work.

Maybe the CI runs pytest-xdist anyway? Or some other pytest version, or other plugins, etc.? Something that would make the order of execution different, e.g. test_vgen.setup_module could be called before the actual tests of test_verify run.

I am stumped. No pytest-xdist plugin is used:

platform linux -- Python 3.13.0, pytest-8.3.3, pluggy-1.5.0

and the tests are run in the correct order:

  ../../../project/cffi/testing/cffi0/test_verify.py ...................s. [ 42%]
  .....s.....s............................................................ [ 46%]
  .....................................................s...s.............s [ 50%]
  s...                                                                     [ 50%]
  ../../../project/cffi/testing/cffi0/test_version.py .....                [ 51%]
  ../../../project/cffi/testing/cffi0/test_vgen.py ...................s... [ 52%]
  ...s.....s.............................................................. [ 56%]
  ...................................................s...s.............ss. [ 60%]
  .F                                

When I can I will add debug cruft to cleanup_tempdir(), maybe somehow it is skipping the module in question.

Another strange thing I noticed in the codebase: there is both testing/cffi0/test_verify.py and testing/cffi1/test_verify1.py. These files echo each other: they have many of the same tests with slight variations. Some of these repeated tests are quite "expensive" on PyPy's nightly testing on windows. For instance:

• test_struct_bad_sized_integer takes 101 seconds
• test_global_const_int_size takes ~75 secs

Still stumped. I used cos to make sure the test is using a unique function call and print the remaining files in the cleanup_tmp() call at the beginning of test_vgen.py, there are no remnants that might trip up the call to verify(). It still seems like something is being cached from test_verify into test_vgen somewhere.

CI was passing so I removed the debug cruft and enabled windows tests. Let's see how long it takes. I am not sure why when using a module name I had to create a unique module name but 🤷. I guess #135 avoided this problem by making all the module names unique.

25 minutes for windows tests, I guess that is acceptable? It could be less if the repetitive tests are removed from cffi0/test_verify and cffi1/test_verify1.py

cffi0 checks the verify() call that was in cffi 0.x.y, while cffi1 checks the new compile() call. This is very different pieces of code, even if they both implement a large common part.

Ahh, makes sense, thanks for clarifying. CI is passing, including enabling windows tests.

Rebased on main and cleaned up history. CC @ngoldbaum. I would consider this ready to merge if CI passes, tests on windows do not take too long, and all the tests (except embedding on windows) run. We should maybe open an issue to revert disabling embedded tests on windows (commit 8be5d8a), although if I understand correctly they were not run in CI before this commit anyway.

I am not sure about the changes in the github CI yaml, each platform has a different CIBW_TEST_COMMAND call.
Linux uses

CIBW_TEST_COMMAND: PYTHONUNBUFFERED=1 python -m pytest ${{ matrix.test_args || '{project}' }}

macOS uses

CIBW_TEST_COMMAND: pip install pip --upgrade; cd {project}; PYTHONUNBUFFERED=1 pytest

and windows uses

CIBW_TEST_COMMAND: ${{ matrix.test_cmd || 'python -m pytest {package}/src/c' }}
which this PR changes to
CIBW_TEST_COMMAND: ${{ matrix.test_cmd || 'python -m pytest {package}' }}

CI test comparison:

So windows is still ~8x slower than linux while running fewer tests 🤷

I wonder if setting PYTHONBUFFERED=1 would make the windows tests even slower.

I personally find the linux version the most readable. The pip upgrade should happen elsewhere probably on Mac.

It looks like test_cmd is unused, so you can delete the matrix.test_cmd bit in the Windows invocation.

I wonder what the original intent of test_cmd was? @nitzmahone?

I wonder what the original intent of test_cmd was?

Just to allow individual Windows matrix entries to override the test command. I'll sometimes enable a larger subset of tests for a small number of targets to verify things that are too slow to do for all Windows targets, esp since we often can't run all targets in parallel. I'll begrudgingly deal with a couple of 20-40 min test runs, but I don't want to sit around waiting for 20 of them running serially 😆.

Running full tests on emulated arches is currently the largest contributor to the overall build time, which is usually why I'll end up turning those back down to just some minimal smoke tests against the built artifact. If it was just those, I wouldn't complain, but they tend to clog up the works because they'll eventually all run at the same time and block shorter targets from running. We could probably get fancy with concurrency groups to try and minimize the total runtime, but that's been a pretty low priority- I usually settle on something like at least one full test run for each platform/arch for release builds. Doing full tests on every Python/platform/arch would be nice, but there's definitely diminishing returns, especially with emulation or slow targets.

Ahh, cool, so let’s leave that part alone.